Unit for and method of image conversion

ABSTRACT

An image conversion unit ( 200 ) for converting a first input image with a first resolution into an output image with a second resolution, comprises a coefficient-calculating means ( 106 ) for calculating a first filter coefficient on basis of pixel values of the first input image and of pixel values of a second input image. The coefficient-calculating means ( 106 ) is arranged to control an adaptive filtering means ( 104 ) for calculating a pixel value of the output image on basis of an input pixel value of the first image and the first filter coefficient

The invention relates to an image conversion unit for converting a firstimage sequence, comprising a first image with a first resolution and asecond image with the first resolution into a second image sequencecomprising a third image with a second resolution, the image conversionunit comprising:

a coefficient-calculating means for calculating a first filtercoefficient on basis of pixel values of the first image;

an adaptive filtering means for calculating a third pixel value of thethird image on basis of a first one of the pixel values of the firstimage and the first filter coefficient.

The invention further relates to a method of converting a first imagesequence, comprising a first image with a first resolution and a secondimage with the first resolution into a second image sequence comprisinga third image with a second resolution, the method comprising:

calculating a first filter coefficient on basis of pixel values of thefirst image; and

calculating a third pixel value of the third image on basis of a firstone of the pixel values of the first image and the first filtercoefficient.

The invention further relates to an image processing apparatuscomprising:

receiving means for receiving a signal corresponding to a first imagesequence; and

the above mentioned image conversion unit for converting the first imagesequence into a second image sequence.

The advent of HDTV emphasizes the need for spatial up-conversiontechniques that enable standard definition (SD) video material to beviewed on high definition (HD) television (TV) displays. Conventionaltechniques are linear interpolation methods such as bi-linearinterpolation and methods using poly-phase low-pass interpolationfilters. The former is not popular in television applications because ofits low quality, but the latter is available in commercially availableICs. With the linear methods, the number of pixels in the frame isincreased, but the high frequency part of the spectrum is not extended,i.e. the perceived sharpness of the image is not increased. In otherwords, the capability of the display is not fully exploited.

Additional to the conventional linear techniques, a number of non-linearalgorithms have been proposed to achieve this up-conversion. Sometimesthese techniques are referred to as content-based or edge dependentspatial up-conversion. Some of the techniques are already available onthe consumer electronics market.

An embodiment of the image conversion unit of the kind described in theopening paragraph is known from the article “New Edge-DirectedInterpolation”, by Xin Li et al., in IEEE Transactions on ImageProcessing, Vol. 10, No 10, October 2001, pp. 1521-1527. In this imageconversion unit, the filter coefficients of an interpolationup-conversion filter are adapted to the local image content. Theinterpolation up-conversion filter aperture uses a fourth orderinterpolation algorithm as specified in Equation 1: $\begin{matrix}{{F_{HD}\left( {{2\left( {i + 1} \right)},{2\left( {j + 1} \right)}} \right)} = {\sum\limits_{k = 0}^{1}{\sum\limits_{l = 0}^{1}{w_{{2k} + l}{F_{SD}\left( {{{2i} + {2k}},{{2j} + {2l}}} \right)}}}}} & (1)\end{matrix}$with F_(HD)(i, j) the luminance values of the HD output pixels,F_(SD)(i, j) the luminance values of the input pixels and w_(i) thefilter coefficients. The filter coefficients are obtained from a largeraperture using a Least Mean Squares (LMS) optimization procedure. In thecited article is explained how the filter coefficients are calculated.The method according to the prior art is also explained in connectionwith FIG. 1A and FIG. 1B. The method aims at interpolating along edgesrather than across them to prevent blurring. The authors make thesensible assumption that edge orientation does not change with scaling.Therefore, the coefficients can be approximated from the SD input imagewithin a local window by using the LMS method.

Although the “New Edge-Directed Interpolation” method according to thecited prior art works relatively well in many image parts, there is aproblem with selecting the appropriate window for the LMS method. Forwindows of size it by in, there are (n−2)(m−2) equations.Experimentally, the inventor found that a window of 4 by 4, whichresults in 4 equations did not lead to a robust up scaling. Betterresults have been obtained using windows of 8 by 8, i.e. with 36equations. Although the up-conversion was more robust, there was alsomore blurring. It is assumed that this is due to the fact that the imagestatistics are not constant over this larger area, which causes thefilter to converge towards a plain averaging filter. To conclude: thereis a conflict that complicates the choice of the window size. On the onehand, because of the robustness the window size has to be large. On theother hand, for constant image statistics the window size has be assmall as possible. Finally, the LMS optimization requires at least thesame number of equations as there are unknown coefficients, which givesa lower bound to the window size.

It is an object of the invention to provide an image conversion unit ofthe kind described in the opening paragraph which is relatively robustwhile the amount of image blur is relatively low.

This object of the invention is achieved in that thecoefficient-calculating means is arranged to calculate the first filtercoefficient on basis of further pixel values of the second image. Inother words the aperture of the coefficient-calculating means isenlarged in the temporal domain rather than in the spatial domain. Theassumption then is that in corresponding -smaller- image parts ofdifferent images, the statistics are more similar than in differentlocations of a -larger- part in the same image. This is particularly tobe expected in the case that the corresponding image parts are takenalong the motion trajectory. So, additional to the assumption that edgeorientation is independent of scale, it is now assumed that edgeorientation is constant over time when corrected for motion. Pixelvalues are luminance values or color values.

Notice that the further pixel values are not applied in the direct pathof processing the input pixels of the first image into output pixels,i.e. the pixels of the third image, but in the control path to determinethe filter coefficients. Combining input pixel values of multiple inputfields into a single output pixel value of a single output image, i.e.frame, is for instance known as de-interlacing. Interlacing is thecommon video broadcast procedure for transmitting the odd and evennumbered image lines alternately. De-interlacing attempts to restore thefull vertical resolution, i.e. make odd and even lines availablesimultaneously for each image. The purpose of de-interlacing is thereduction of alias in successive fields. However a purpose of the imageconversion unit according to the present invention is to increase theresolution of input images on basis of respective input images. This isdone by means of a spatial filter which is adapted to edges in order tolimit the amount of blur which would arise without the adaptation to theedges. The spatial filter in controlled by means of filter coefficientswhich are determined on basis of multiple input images.

An embodiment of the image conversion unit according to the invention isarranged to acquire the pixel values of the first image from a firstpart of the first image and the further pixel values of the second imagefrom a second part of the second image, with the first part and thesecond part spatially corresponding. An advantage of this embodiment isthat it is relatively simple. Acquisition of the appropriate pixels fromthe second image is straight forward without additional calculations.Temporarily storage of a number of pixel values of the second image isrequired.

An embodiment of the image conversion unit according to the invention isarranged to acquire the pixel values of the first image from a firstpart of the first image and the further pixel values of the second imagefrom a second part of the second image, with the first part and thesecond part at a motion trajectory. Motion vectors have to be providedby means of a motion estimator. These motion vectors describe therelation between the first part and the second part. An advantage ofthis embodiment is that the images of the second sequence, i.e. theoutput images, are relatively sharp.

In an embodiment of the image conversion unit according to the inventionthe coefficient-calculating means is arranged to calculate the firstfilter coefficient by means of an optimization algorithm. Preferably theoptimization algorithm is a Least Mean Square algorithm. An LMSalgorithm is relatively simple and robust.

It is a further object of the invention to provide a method of the kinddescribed in the opening paragraph which is relatively robust while theamount of image blur is relatively low.

This object of the invention is achieved in that the first filtercoefficient is calculated on basis of further pixel values of the secondimage.

It is a further object of the invention to provide an image processingapparatus of the kind described in the opening of which the imageconversion unit is relatively robust while the amount of image blur isrelatively low.

This object of the invention is achieved in that thecoefficient-calculating means of the image processing apparatus isarranged to calculate the first filter coefficient on basis of furtherpixel values of the second image. The image processing apparatusoptionally comprises a display device for displaying the second image.The image processing apparatus might e.g. be a TV, a set top box, a VCR(Video Cassette Recorder) player or a DVD (Digital Versatile Disk)player. Modifications of image conversion unit and variations thereofmay correspond to modifications and variations thereof of the method andof the image processing apparatus described.

These and other aspects of the image conversion unit, of the method andof the image processing apparatus according to the invention will becomeapparent from and will be elucidated with respect to the implementationsand embodiments described hereinafter and with reference to theaccompanying drawings, wherein:

FIG. 1A schematically shows an embodiment of the image conversion unitaccording to the prior art;

FIG. 1B schematically shows a number of pixels to explain the methodaccording to the prior art;

FIG. 2A schematically shows two images to explain an embodiment of themethod according to the invention;

FIG. 2B schematically shows two images to explain an alternativeembodiment of the method according to the invention;

FIG. 2C schematically shows an embodiment of the image conversion unitaccording to the invention;

FIG. 3A schematically shows an SD input image;

FIG. 3B schematically shows the SD input image of FIG. 3A on whichpixels are added in order to increase the resolution;

FIG. 3C schematically shows the image of FIG. 3B after being rotatedover 45 degrees;

FIG. 3D schematically shows an HD output image derived from the SD inputimage of FIG. 3A; and

FIG. 4 schematically shows an embodiment of the image processingapparatus according to the invention.

Same reference numerals are used to denote similar parts throughout thefigures.

FIG. 1A schematically shows an embodiment of the image conversion unit100 according to the prior art. The image conversion unit 100 isprovided with standard definition (SD) images at the input connector 108and provides high definition (HD) images at the output connector 110.The image conversion unit 100 comprises:

A pixel acquisition unit 102 which is arranged to acquire a first set ofpixel values of pixels 1-4 (See FIG. 1B) in a first neighborhood of aparticular location within a first one of the SD input images whichcorresponds with the location of an HD output pixel and is arranged toacquire a second set of pixel values of pixels 1-16 in a secondneighborhood of the particular location within the first one of the SDinput images;

A filter coefficient-calculating unit 106, which is arranged tocalculate filter coefficients on basis of the first set of pixel valuesand the second set of pixel values. In other words, the filtercoefficients are approximated from the SD input image within a localwindow. This is done by using a Least Mean Squares (LMS) method which isexplained in connection with FIG. 1B.

An adaptive filtering unit 104 for calculating the pixel value of the HDoutput pixel on basis of the first set of pixel values and the filtercoefficients as specified in Equation 1. Hence the filtercoefficient-calculating unit 106 is arranged to control the adaptivefiltering unit 104.

FIG. 1B schematically shows a number of pixels 1-16 of an SD input imageand one H) pixel of an HD output image, to explain the method accordingto the prior art. The HD output pixel is interpolated as a weightedaverage of 4 pixel values of pixels 1-4. That means that the luminancevalue of the HD output pixel F_(HD) results as a weighted sum of theluminance values of its 4 SD neighboring pixels:F _(HD) =w ₁ F _(SD)(1)+w ₂ F _(SD)(2)+w ₃ F _(SD)(3)+w ₄ F _(SD)(4),(2)where F_(SD)(1) to F_(SD)(4) are the pixel values of the 4 SD inputpixels 1-4 and w₁ to w₄ are the filter coefficients to be calculated bymeans of the LMS method. The authors of the cited article in which theprior art method is described, make the sensible assumption that edgeorientation does not change with scaling. The consequence of thisassumption is that the optimal filter coefficients are the same as thoseto interpolate, on the standard resolution grid:

Pixel 1 from 5, 7, 11, and 4 (that means that pixel 1 can be derivedfrom its 4 neighbors)

Pixel 2 from 6, 8, 3, and 12

Pixel 3 from 9, 2, 13, and 15

Pixel 4 from 1, 10, 14, and 16

This gives a set of 4 linear equations from which with theLSM-optimization the optimal 4 filter coefficients to interpolate the HDoutput pixel are found.

Denoting M as the pixel set, on the SD -grid, used to calculate the 4weights, the Means Square Error (MSE) over set M in the optimization canbe written as the sum of squared differences between original SD-pixelsF_(SD) and interpolated SD-pixels F_(SI): $\begin{matrix}{{MSE} = {\sum\limits_{{F_{SD}{({i,j})}} \in \quad M}^{\quad}\left( {{F_{SD}\left( {{{2i} + 2},{{2j} + 2}} \right)} - {F_{SI}\left( {{{2i} + 2},{{2j} + 2}} \right)}} \right)^{2}}} & (3)\end{matrix}$Which in matrix formulation becomes: $\begin{matrix}{{MSE} = {{\overset{\rightarrow}{y} - {\overset{\rightarrow}{w}\quad C}}}^{2}} & (4)\end{matrix}$Here {right arrow over (y)} contains the SD-pixels in M (pixelF_(SD)(1,1) to F_(SD)(1,4), F_(SD)(2,1) to F_(SD)(2,4), F_(SD)(3,1) toF_(SD)(3,4), F_(SD)(4,1) to F_(SD)(4,4) and C is a 4×M² matrix whosek^(th) row is composed of the weighted sum of the four diagonalSD-neighbors of each SD-pixels in {right arrow over (y)}.

The weighted sum of each row describes a pixel F_(SI), as used inEquation 3. To find the minimum MSE, i.e. LMS, the derivation of MSEover {right arrow over (w)} is calculated: $\begin{matrix}{\frac{\partial({MSE})}{\partial w} = 0} & (5) \\{{{{- 2}\overset{\rightarrow}{y}\quad C} + {2\overset{\rightarrow}{w}\quad C^{2}}} = 0} & (6) \\{\overset{\rightarrow}{w} = {\left( {C^{T}C} \right)^{- 1}\left( {C^{T}\overset{\rightarrow}{y}} \right)}} & (7)\end{matrix}$By solving Equation 7 the filter coefficients are found and by usingEquation 2 the pixel values of the HD output pixels can be calculated.

In this example a window of 4 by 4 pixels is used for the calculation ofthe filter coefficients. An LMS optimization on a larger window, e.g. 8by 8 instead of 4 by 4 gives better results.

FIG. 2A schematically shows two SD input images 202, 204 to explain anembodiment of the method according to the invention. Each of the two SDinput images 202, 204 comprises a number of pixels, e.g. 210-220 whichare indicated with X-signs. Suppose that an HD output pixel has to becalculated. The location corresponding to this HD output pixel isindicated in a first one of the input images 202. For the calculation ofa first filter coefficient, e.g. with which SD input pixel 212 has to bemultiplied, a set of equations has to be solved as explained inconnection with FIG. 1B. The known components of these equationscorrespond with pixel values e.g. 210-215 taken from a first part 206 ofthe first one of the input images 202, but also with pixel values e.g.216-220 taken from a second part 208 of a second one of the input images204. Preferably the pixel values which are used to determine he firstfilter coefficient of a particular pixel 212 are acquired from the localneighborhood. That means that the pixels which are connected to theparticular pixel 212 are applied, e.g. the upper, the lower, the right,the left and the diagonal pixels. The pixel values of the second imageare also acquired from a local neighborhood which corresponds to thelocal neighborhood in the first image. The first part 206 of the firstone of the input images 202 and the second part 208 of the second one ofthe input images 204 are spatially corresponding. That means that allrespective pixels of the first part 206 have the same coordinates as thecorresponding pixels of the second part 208. That is not the case withthe images parts 206 and 222 as depicted in FIG. 2B.

FIG. 2B schematically shows two images 202, 204 to explain analternative embodiment of the method according to the invention. Thefirst part 206 of the first one of the input images 202 and the thirdpart 222 of the second one of the input images 204 are located at amotion trajectory. The relation between the first part 206 and the thirdpart 222 is determined by the motion vector 230 which has beencalculated by means of a motion estimator. This motion estimator mightbe the motion estimator as described in the article “True-MotionEstimation with 3-D Recursive Search Block Matching” by G. de Haan et.al. in IEEE Transactions on circuits and systems for video technology,vol. 3, no. 5, October 1993, pages 368-379. In this case the respectivepixels of the two image parts correspond to substantially equal picturecontent although there was movement of objects in the scene beingimaged.

FIG. 2C schematically shows an embodiment of the image conversion unit200 according to the invention. The image conversion unit 200 isprovided with standard definition (SD) images at the input connector 108and provides high definition (HD) images at the output connector 110.The SD input images have pixel matrices as specified in CCIR-601, e.g.625*720 pixels or 525*720 pixels. The HD output images have pixelmatrices with e.g. twice or one-and-a-halve times the number of pixelsin horizontal and vertical direction. The image conversion unit 200comprises:

A memory device for storage of a number of pixels of a number of SDinput images.

A pixel acquisition unit 102 which is arranged to acquire:

a first set of pixel values of pixels from a first one of the SD inputimages in a first neighborhood of a particular location within the firstSD input image, which corresponds with the location of the output pixelHD.

a second set of pixel values of pixels from the first SD input image ina second neighborhood of the particular location;

a third set of pixel values of pixels from a second one of the SD inputimages in a third neighborhood of the particular location;

an optional fourth set of pixel values of pixels from a third one of theSD input images in a fourth neighborhood of the particular location.

A filter coefficient-calculating unit 106 which is arranged to calculatefilter coefficients on basis of the first, second, third and optionallyfourth set of pixel values. In other words, the filter coefficients areapproximated from the SD input images within a local window located inthe first SD input image and the window extending to the second SD inputimage and optionally to the third SD input image. Preferably the secondSD input image and the third SD input image are respectively precedingand succeeding the first SD input image in the sequence of SD inputimages. The approximation of the filter coefficients is done by using aLeast Mean Squares (LMS) method which is explained in connection withFIG. 1B, FIG. 2A and FIG. 2B; and

An adaptive filtering unit 104 for calculating a pixel value of an HDoutput image on basis of the second set of pixel values. The HD outputpixel is calculated as the weighted sum of the pixel values of the firstset of pixel values.

The image conversion unit 200 optionally comprises an input connector114 for providing motion vectors to be applied by the pixel acquisitionunit 102 for the acquisition of pixel values in the succeeding SD inputimages of the SD input image sequence, which are on respective motiontrajectories, as explained in connection with FIG. 2B.

The number of pixels acquired in the neighborhood, i.e. the window size,might be even or odd, e.g. 4*4 or 5*5 respectively. Besides that theshape of the window does not have to be rectangular. Also the number ofpixels acquired from the first image and the number of pixels acquiredfrom the second image does not have to be mutually equal.

The pixel acquisition unit 102, the filter coefficient-calculating unit106 and the adaptive filtering unit 104 may be implemented using oneprocessor. Normally, these functions are performed under control of asoftware program product. During execution, normally the softwareprogram product is loaded into a memory, like a RAM, and executed fromthere. The program may be loaded from a background memory, like a ROM,hard disk, or magnetically and/or optical storage, or may be loaded viaa network like Internet. Optionally an application specific integratedcircuit provides the disclosed functionality.

To convert an SD input image into an HD output image a number ofprocessing steps are needed. By means of FIGS. 3A-3D these processingsteps are explained. FIG. 3A schematically shows an SD input image; FIG.3D schematically shows an HD output image derived from the SD inputimage of FIG. 3A and FIGS. 3B and 3C schematically show intermediateresults.

FIG. 3A schematically shows an SD input image. Each X-sign correspondwith a respective pixel.

FIG. 3B schematically shows the SD input image of FIG. 3A on whichpixels are added in order to increase the resolution. The added pixelsare indicated with +-signs. These added pixels are calculated by meansof interpolation of the respective diagonal neighbors. The filtercoefficients for the interpolation are determined as described inconnection with FIG. 2B.

FIG. 3C schematically shows the image of FIG. 3B after being rotatedover 45 degrees. The same image conversion unit 200 as being applied tocalculate the image as depicted in FIG. 3B on basis of FIG. 3A can beused to calculate the image as shown in FIG. 3D on basis of the image asdepicted in FIG. 3B. That means that new pixel values are calculated bymeans of interpolation of the respective diagonal neighbors. Notice thata first portion of these diagonal neighbors (indicated with X-signs)correspond to the original pixel values of the SD input image and that asecond portion of these diagonal neighbors (indicated with +-signs)correspond to pixel values which have been derived from the originalpixel values of the SD input image by means of interpolation.

FIG. 3D schematically shows the final HD output image. The pixels thathave been added in the last conversion step are indicated with o-signs.

FIG. 4 schematically shows an embodiment of the image processingapparatus 400 according to the invention, comprising:

Receiving means 402 for receiving a signal representing SD images. Thesignal may be a broadcast signal received via an antenna or cable butmay also be a signal from a storage device like a VCR (Video CassetteRecorder) or Digital Versatile Disk (DVD). The signal is provided at theinput connector 408;

The image conversion unit 404 as described in connection with FIG. 2B;and

A display device 406 for displaying the HD output images of the imageconversion unit 200. This display device 406 is optional.

The image processing apparatus 400 might e.g. be a TV. Alternatively theimage processing apparatus 400 does not comprise the optional displaydevice but provides HD images to an apparatus that does comprise adisplay device 406. Then the image processing apparatus 400 might bee.g. a set top box, a satellite-tuner, a VCR player or a DVD player. Butit might also be a system being applied by a film-studio or broadcaster.

It should be noted that the above-mentioned embodiments illustraterather than limit the invention and that those skilled in the art willbe able to design alternative embodiments without departing from thescope of the appended claims. In the claims, any reference signs placedbetween parentheses shall not be constructed as limiting the claim. Theword ‘comprising’ does not exclude the presence of elements or steps notlisted in a claim. The word “a” or “an” preceding an element does notexclude the presence of a plurality of such elements. The invention canbe implemented by means of hardware comprising several distinct elementsand by means of a suitable programmed computer. In the unit claimsenumerating several means, several of these means can be embodied by oneand the same item of hardware.

1. An image conversion unit (200) for converting a first image sequence,comprising a first image with a first resolution and a second image withthe first resolution into a second image sequence comprising a thirdimage with a second resolution, the image conversion unit (200)comprising: a coefficient-calculating means (106) for calculating afirst filter coefficient on basis of pixel values of the first image; anadaptive filtering means (104) for calculating a third pixel value ofthe third image on basis of a first one of the pixel values of the firstimage and the first filter coefficient, characterized in that thecoefficient-calculating means (106) is arranged to calculate the firstfilter coefficient on basis of further pixel values of the second image.2. An image conversion unit (200) as claimed in claim 1, characterizedin that the image conversion unit (200) is arranged to acquire the pixelvalues of the first image from a first part of the first image and thefurther pixel values of the second image from a second part of thesecond image, with the first part and the second part spatiallycorresponding.
 3. An image conversion unit (200) as claimed in claim 1,characterized in that the image conversion unit (200) is arranged toacquire the pixel values of the first image from a first part of thefirst image and the further pixel values of the second image from asecond part of the second image, with the first part and the second partat a motion trajectory.
 4. An image conversion unit (200) as claimed inclaim 1, characterized in that the coefficient-calculating means (106)is arranged to calculate the first filter coefficient by means of anoptimization algorithm.
 5. A method of converting a first imagesequence, comprising a first image with a first resolution and a secondimage with the first resolution into a second image sequence comprisinga third image with a second resolution, the method comprising:calculating a first filter coefficient on basis of pixel values of thefirst image; and calculating a third pixel value of the third image onbasis of a first one of the pixel values of the first image and thefirst filter coefficient, characterized in that the first filtercoefficient is calculated on basis of further pixel values of the secondimage.
 6. An image processing apparatus (400) comprising: receivingmeans (402) for receiving a signal corresponding to a first imagesequence; and the image conversion unit (404) for converting the firstimage sequence into a second image sequence, as claimed in claim
 1. 7.An image processing apparatus (400) as claimed in claim 7, characterizedin further comprising a display device (406) for displaying the secondimage sequence.
 8. An image processing apparatus (400) as claimed inclaim 8, characterized in that it is a TV.